How a lack of transparency is damaging sample quality

Editor’s note: Courtney Williams is executive director of quality at computer software firm Lucid, New Orleans. 

The need for transparency is a core trend we see across many industries. As consumers, we value transparency in the brands we buy. As employees, we value transparency from our employers. So naturally – as sample buyers – we value transparency in our suppliers. We should expect it.

But are we getting that transparency from panel suppliers? 

A changing structure

For a few years now there has been a trend for clients to bring their market research projects in-house. Most of the major panel providers have adjusted their marketing and communications to include directly targeting end-clients, and there has been a movement of agency researchers to client-side teams. 

With this surge of in-house market research there has also been an increase in the practice of panel companies blending sample with unknown suppliers. This happens for many reasons – the main one being feasibility challenges. In many cases, those buying sample on behalf of in-client teams have an informed knowledge of how the panel industry works. They may know that their sample, delivered by one of the major panel providers, is most likely being sourced from several different panels. 

This certainly isn’t a secret but more often than not the buyer doesn’t know where their sample comes from. For newer researchers on client teams without past experience with panel companies, the fact that their sample is sourced from several panels may come as a complete surprise – and that’s problematic. 

Transparency 

The panel supplier cartel still runs on an old model, one that lacks transparency. To be fair, it didn’t start this way. Today, most providers are simply unable (or unwilling) to single-source most research projects. Running projects without transparency is doing immense damage to the industry and potentially to your data. The source of your respondents is important; the way a panel company recruits, retains, engages and incentivizes sample is proven to affect data composition and quality. However, when you anonymize and aggregate multiple sources into a single brand name, the buyer is left without any insight as to the potential structural difference of their sample. 

Client-side researchers are well aware of how technology is changing the way researchers collect and analyze data. Technology has democratized our access to data – AI and automation, for example, have shortened the time in-field and in-analysis. Programmatic technology opened up sample buying in a way our industry has never seen before. A marketplace in the sample industry allows the owner of the sample to sell directly to the buyer at the price they want. There’s no hidden blending of different sources, no price fixing and buyers can choose the supplier or suppliers that consistently meet their needs rather than being at the mercy of their panel provider. 

Moving forward 

Transparency is a serious issue in the sample industry. It affects trust, the quality of data and healthy competition. Many big panel suppliers discuss data quality and respondent engagement but they often frame it in terms of questionnaire quality. While highly important, this is only part of the conversation. This one-sided focus on questionnaires places all responsibility on the researcher, ignoring the fact that suppliers themselves are not completely in control of the sample they provide to clients. With the transparency provided by a programmatic marketplace, full (measurable) accountability of sample quality is placed on each individual panel supplier – providing more incentive to improve the processes and data quality. 

This is an industry-wide problem, which means that we must all take the initiative to solve it. It’s time to reframe our conversations and move away from decade-old topics like survey length and design. If a researcher is still putting 40-minute surveys into field – despite resounding acceptance of short surveys and mobile agnostic surveys – they’re not likely to change their methods. Yes, the topic is important but our continued focus on it suggests that survey length and design are the only key issues. We need to facilitate discussions about more progressive issues like quality and transparency.

Additionally, client-side researchers need to ensure they’re asking suppliers the right questions: Where is my sample coming from? How are those third-party suppliers engaging their respondents? What data is coming from which supplier? In B2C categories, consumers can create change through the power of their voice and often it can lead to a wholesale change in business practices. As client-side researchers and consumers, transparency only happens if you demand it strongly enough. Consistently ask your supplier the right questions; they should be able to provide you with the information you need to make informed choices on your sample. If they refuse to provide the transparency you need, you have a choice to source elsewhere. 

Elevating quality 

I believe that in a free and open market everyone wins. Buyers should be able to make well-informed decisions about how they fuel their research and suppliers should be able to decide to whom they send their sample. It is through transparency and the resulting accountability that we open the conversation and take another step toward elevating quality in our industry.